Pictorial symbol prediction

ABSTRACT

Symbol prediction can be implemented using a multi-task system trained for different tasks. The tasks may include a single symbol prediction, symbol category prediction, and symbol subcategory prediction. Categories of symbols can be generated by clustering sets of training data using a clustering scheme.

PRIORITY CLAIM

This application is a non-provisional application which claims thebenefit of priority to U.S. Provisional Application Ser. No. 62/526,906,filed Jun. 29, 2018; and U.S. Provisional Application Ser. No.62/599,640, filed Dec. 15, 2017, the contents of which are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to machinelearning and, more particularly, but not by way of limitation, tomachine learning-based symbol suggestions.

BACKGROUND

Emojis are ideograms (e.g., pictures, symbols) that can be used toconcisely describe an idea or complement nearby text with an emotion,concept, or a hard to describe subtle aspect. How emojis are used andunderstood can depend on the groups of people using them. Due to theirability to describe difficult and subtle aspects of an idea, emojisuggestions generated by machine learning schemes can often benon-relevant or otherwise inaccurate.

BRIEF DESCRIPTION OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to the FIG.(“FIG.”) number in which that element or act is first introduced.

FIG. 1 is a block diagram showing an example messaging system forexchanging data (e.g., messages and associated content) over a network.

FIG. 2 is a block diagram illustrating further details regarding themessaging system of FIG. 1, according to example embodiments.

FIG. 3 is a schematic diagram illustrating data which may be stored in adatabase of a messaging server system, according to certain exampleembodiments.

FIG. 4 is a schematic diagram illustrating a structure of a message,according to some embodiments, generated by a messaging clientapplication for communication.

FIG. 5 is a schematic diagram illustrating an example access-limitingprocess, in terms of which access to content (e.g., an ephemeralmessage, and associated multimedia payload of data) or a contentcollection (e.g., an ephemeral message story) may be time-limited (e.g.,made ephemeral), according to some example embodiments.

FIG. 6 illustrates example functional components of a pictorial symbolsystem, according to some example embodiments.

FIG. 7 illustrates example functional engines of the task system,according to some example embodiments.

FIG. 8A shows a flow diagram of a method for pictorial symbolclassification and content selection, according to some exampleembodiments.

FIG. 8B shows an example flow diagram of a method for generatingclassifications, according to some example embodiments.

FIG. 8C shows an example flow diagram of a method for generatingclassifications, according to some example embodiments.

FIG. 9 shows a flow diagram of a method of semantically generatingpictorial symbols using a clustering scheme, according to some exampleembodiments.

FIG. 10 shows an example architecture of the multi-task engine,according to some example embodiments.

FIG. 11 shows example internal functional engines of a multimodalsystem, according to some example embodiments.

FIG. 12 shows a flow diagram of a method for implementing emojisuggestions from multimodal messages, according to some exampleembodiments.

FIG. 13 shows a flow diagram of a method for multimodal suggestions,according to some example embodiments.

FIG. 14 shows an example flow diagram of a method for displaying symbolresults, according to some example embodiments.

FIG. 15 shows an example user interface for pictorial symbol prediction,according to some example embodiments.

FIG. 16 shows an example user interface for pictorial symbol prediction,according to some example embodiments.

FIG. 17 shows an example set of pictorial symbols, according to someexample embodiments.

FIG. 18 is a block diagram illustrating a representative softwarearchitecture, which may be used in conjunction with various hardwarearchitectures herein described.

FIG. 19 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques,instruction sequences, and computing machine program products thatembody illustrative embodiments of the disclosure. In the followingdescription, for the purposes of explanation, numerous specific detailsare set forth in order to provide an understanding of variousembodiments of the inventive subject matter. It will be evident,however, to those skilled in the art, that embodiments of the inventivesubject matter may be practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures, andtechniques are not necessarily shown in detail.

As discussed above, symbol suggestion is difficult due to subtlelinkages between given symbols and how users apply the symbols. To thisend, a pictorial symbol system is configured to receive messages (e.g.,text/caption data, video data with audio voice data) and generaterelevant suggestions for display to the user. In some exampleembodiments, the pictorial symbol system includes a task sub-system thatis trained to generate symbol predictions at different granularities or“tasks”. For example, the task sub-system can be trained on a firstsingle symbol prediction task, a second symbol category prediction task,and a third sub-category symbol prediction task.

In some example embodiments, the symbols are grouped into pre-setcategories and sub-categories, such as Unicode emoji categories andsub-categories. (Unicode is a computing standard to reference encodingand handling of text items and symbols.) FIG. 17 shows examples ofemojis 1702-1740. The task sub-system is able to capture linkagesbetween a given caption and individual single emojis, and also betweenthe given caption and the categories and subcategories. This isbeneficial because if the first task prediction is not sufficientlyaccurate, the task sub-system can fall back on broader categories andsub-categories, which may yield higher accuracy related to a givenuser's caption.

In some example embodiments, the task sub-system is trained onsemantically generated categories that capture linkages missed by thepre-set categories and sub-categories. The semantically generatedcategories can be generated using a clustering scheme (e.g., k-means).In some embodiments, the task sub-system falls back on semanticallygenerated category predictions if the symbol category predictions arenot accurate or do not meet a pre-specified threshold.

In some example embodiments, the pictorial symbol system includes amultimodal sub-system that is trained to generate symbol predictionsbased on keywords in audio data recorded by a user's client device. Themultimodal sub-system can implement a machine learning scheme, such asrandom forest, that is trained on publicly available network posts thatdo not have music and are in the primary language of the client device.In some example embodiments, the primary language is identified bydetermining the keyboard language setting of the user's client device.In some example embodiments, the primary language is determined byapplying a language recognition scheme to audio data recorded by theuser of the client device.

FIG. 1 is a block diagram showing an example messaging system 100 forexchanging data (e.g., messages and associated content) over a network.The messaging system 100 includes multiple client devices 102, each ofwhich hosts a number of applications including a messaging clientapplication 104. Each messaging client application 104 iscommunicatively coupled to other instances of the messaging clientapplication 104 and a messaging server system 108 via a network 106(e.g., the Internet). In various embodiments, virtual machine learningcan be used by messaging client application 104 and/or image processingsystem 116 to analyze images sent within the messaging system 100, andto use this analysis to provide features within the messaging system100.

Accordingly, each messaging client application 104 is able tocommunicate and exchange data with another messaging client application104 and with the messaging server system 108 via the network 106. Thedata exchanged between messaging client applications 104, and between amessaging client application 104 and the messaging server system 108,include functions (e.g., commands to invoke functions) as well aspayload data (e.g., text, audio, video, or other multimedia data).

The messaging server system 108 provides server-side functionality viathe network 106 to a particular messaging client application 104. Whilecertain functions of the messaging system 100 are described herein asbeing performed by either a messaging client application 104 or by themessaging server system 108, it will be appreciated that the location ofcertain functionality within either the messaging client application 104or the messaging server system 108 is a design choice. For example, itmay be technically preferable to initially deploy certain technology andfunctionality within the messaging server system 108, but to latermigrate this technology and functionality to the messaging clientapplication 104 where a client device 102 has a sufficient processingcapacity.

The messaging server system 108 supports various services and operationsthat are provided to the messaging client application 104. Suchoperations include transmitting data to, receiving data from, andprocessing data generated by the messaging client application 104. Thisdata may include message content, client device information, geolocationinformation, media annotation and overlays, message content persistenceconditions, image search, social network information, and live eventinformation, as examples, some of which rely on information generated byanalyzing images sent through the messaging system 100. Data exchangeswithin the messaging system 100 are invoked and controlled throughfunctions available via user interfaces (UIs) of the messaging clientapplication 104.

Turning now specifically to the messaging server system 108, anapplication programming interface (API) server 110 is coupled to, andprovides a programmatic interface to, an application server 112. Theapplication server 112 is communicatively coupled to a database server118, which facilitates access to a database 120 in which is stored dataassociated with messages processed by the application server 112. Insome embodiments, databases 120 may also store results of imageprocessing, or details of various trained and untrained support vectormachines that may be used by messaging server system 108.

The API server 110 receives and transmits message data (e.g., commandsand message payloads) between the client device 102 and the applicationserver 112. Specifically, the API server 110 provides a set ofinterfaces (e.g., routines and protocols) that can be called or queriedby the messaging client application 104 in order to invoke functionalityof the application server 112. The API server 110 exposes variousfunctions supported by the application server 112, including accountregistration; login functionality; the sending of messages, via theapplication server 112, from a particular messaging client application104 to another messaging client application 104; the sending of mediafiles (e.g., images or video) from a messaging client application 104 toa messaging server application 114 for possible access by anothermessaging client application 104; the setting of a collection of mediadata (e.g., a story); the retrieval of such collections; the retrievalof a list of friends of a user of a client device 102; the retrieval ofmessages and content; the addition and deletion of friends to and from asocial graph; the location of friends within the social graph; andapplication events (e.g., relating to the messaging client application104).

The application server 112 hosts a number of applications andsubsystems, including the messaging server application 114, an imageprocessing system 116, a social network system 122, and a pictorialsymbol system 150, according to some example embodiments. The messagingserver application 114 implements a number of message processingtechnologies and functions, particularly related to the aggregation andother processing of content (e.g., textual and multimedia content)included in messages received from multiple instances of the messagingclient application 104. As will be described in further detail, the textand media content from multiple sources may be aggregated intocollections of content (e.g., called stories or galleries). Thesecollections are then made available, by the messaging server application114, to the messaging client application 104. Other processor- andmemory-intensive processing of data may also be performed server-side bythe messaging server application 114, in view of the hardwarerequirements for such processing.

The application server 112 also includes the image processing system116, which is dedicated to performing various image processingoperations, typically with respect to images or video received withinthe payload of a message at the messaging server application 114.

The social network system 122 supports various social networkingfunctions and services, and makes these functions and services availableto the messaging server application 114. To this end, the social networksystem 122 maintains and accesses an entity graph (e.g., entity graph304 in FIG. 3) within the database 120. Examples of functions andservices supported by the social network system 122 include theidentification of other users of the messaging system 100 with whom aparticular user has relationships or whom the particular user is“following”, and also the identification of other entities and interestsof a particular user.

Further, although FIG. 1 shows the pictorial symbol system 150integrated into the application server 112, in some example embodiments,the pictorial symbol system 150 is integrated entirely within messagingclient application 104. Further, in some example embodiments, some ofthe engines of the pictorial symbol system 150 are executed on a server(e.g., application server 112) and some of the engines of the pictorialsymbol system 150 are executed from the client device 102 (e.g., as partof the messaging client application 104). For example, an instance ofthe machine learning system (discussed below) may be trained to create aclassifier model. The classifier model data can then be transferred toone or more client devices 102. On the one or more client devices 102,another instance of the machine learning system may apply the receivedclassifier model to classify voice data in a snap and recommend an emojion the client device 102 without connecting to the application server112 to generate the suggestion.

The application server 112 is communicatively coupled to a databaseserver 118, which facilitates access to a database 120 in which isstored data associated with messages processed by the messaging serverapplication 114.

FIG. 2 is block diagram illustrating further details regarding themessaging system 100, according to example embodiments. Specifically,the messaging system 100 is shown to comprise the messaging clientapplication 104 and the application server 112, which in turn embody anumber of subsystems, namely an ephemeral timer system 202, a collectionmanagement system 204, an annotation system 206, and a pictorial symbolsystem or pictorial symbol system 150. The pictorial symbol system 150is discussed in further detail below.

The ephemeral timer system 202 is responsible for enforcing thetemporary access to content permitted by the messaging clientapplication 104 and the messaging server application 114. To this end,the ephemeral timer system 202 incorporates a number of timers that,based on duration and display parameters associated with a message orcollection of messages (e.g., a story), selectively display and enableaccess to messages and associated content via the messaging clientapplication 104. Further details regarding the operation of theephemeral timer system 202 are provided below.

The collection management system 204 is responsible for managingcollections of media (e.g., collections of text, image, video, and audiodata). In some examples, a collection of content (e.g., messages,including images, video, text, and audio) may be organized into an“event gallery” or an “event story”. Such a collection may be madeavailable for a specified time period, such as the duration of an eventto which the content relates. For example, content relating to a musicconcert may be made available as a “story” for the duration of thatmusic concert. The collection management system 204 may also beresponsible for publishing an icon that provides notification of theexistence of a particular collection to the user interface of themessaging client application 104.

The collection management system 204 furthermore includes a curationinterface 208 that allows a collection manager to manage and curate aparticular collection of content. For example, the curation interface208 enables an event organizer to curate a collection of contentrelating to a specific event (e.g., delete inappropriate content orredundant messages). Additionally, the collection management system 204employs machine vision (or image recognition technology) and contentrules to automatically curate a content collection. In certainembodiments, compensation may be paid to a user for inclusion ofuser-generated content into a collection. In such cases, the curationinterface 208 operates to automatically make payments to such users forthe use of their content.

The annotation system 206 provides various functions that enable a userto annotate or otherwise modify or edit media content associated with amessage. For example, the annotation system 206 provides functionsrelated to the generation and publishing of media overlays for messagesprocessed by the messaging system 100. The annotation system 206operatively supplies a media overlay (e.g., a geofilter or filter) tothe messaging client application 104 based on a geolocation of theclient device 102. In another example, the annotation system 206operatively supplies a media overlay to the messaging client application104 based on other information, such as social network information ofthe user of the client device 102. A media overlay may include audio andvisual content and visual effects. Examples of audio and visual contentinclude pictures, texts, logos, animations, and sound effects. Anexample of a visual effect includes color overlaying. The audio andvisual content or the visual effects can be applied to a media contentitem (e.g., a photo) at the client device 102. For example, the mediaoverlay includes text that can be overlaid on top of a photographgenerated by the client device 102. In another example, the mediaoverlay includes an identification of a location (e.g., Venice Beach), aname of a live event, or a name of a merchant (e.g., Beach CoffeeHouse). In another example, the annotation system 206 uses thegeolocation of the client device 102 to identify a media overlay thatincludes the name of a merchant at the geolocation of the client device102. The media overlay may include other indicia associated with themerchant. The media overlays may be stored in the database 120 andaccessed through the database server 118.

In one example embodiment, the annotation system 206 provides auser-based publication platform that enables users to select ageolocation on a map, and upload content associated with the selectedgeolocation. The user may also specify circumstances under whichparticular content should be offered to other users. The annotationsystem 206 generates a media overlay that includes the uploaded contentand associates the uploaded content with the selected geolocation.

In another example embodiment, the annotation system 206 provides amerchant-based publication platform that enables merchants to select aparticular media overlay associated with a geolocation via a biddingprocess. For example, the annotation system 206 associates the mediaoverlay of a highest-bidding merchant with a corresponding geolocationfor a predefined amount of time

FIG. 3 is a schematic diagram illustrating data 300 which may be storedin the database 120 of the messaging server system 108, according tocertain example embodiments. While the content of the database 120 isshown to comprise a number of tables, it will be appreciated that thedata could be stored in other types of data structures (e.g., as anobject-oriented database).

The database 120 includes message data stored within a message table314. An entity table 302 stores entity data, including an entity graph304. Entities for which records are maintained within the entity table302 may include individuals, corporate entities, organizations, objects,places, events, etc. Regardless of type, any entity regarding which themessaging server system 108 stores data may be a recognized entity. Eachentity is provided with a unique identifier, as well as an entity typeidentifier (not shown).

The entity graph 304 furthermore stores information regardingrelationships and associations between or among entities. Suchrelationships may be social, professional (e.g., work at a commoncorporation or organization), interested-based, or activity-based,merely for example.

The database 120 also stores annotation data, in the example form offilters, in an annotation table 312. Filters for which data is storedwithin the annotation table 312 are associated with and applied tovideos (for which data is stored in a video table 310) and/or images(for which data is stored in an image table 308). Filters, in oneexample, are overlays that are displayed as overlaid on an image orvideo during presentation to a recipient user. Filters may be of varioustypes, including user-selected filters from a gallery of filterspresented to a sending user by the messaging client application 104 whenthe sending user is composing a message. Other types of filters includegeolocation filters (also known as geo-filters) which may be presentedto a sending user based on geographic location. For example, geolocationfilters specific to a neighborhood or special location may be presentedwithin a user interface by the messaging client application 104, basedon geolocation information determined by a Global Positioning System(GPS) unit of the client device 102. Another type of filter is a datafilter, which may be selectively presented to a sending user by themessaging client application 104, based on other inputs or informationgathered by the client device 102 during the message creation process.Examples of data filters include a current temperature at a specificlocation, a current speed at which a sending user is traveling, abattery life for a client device 102, or the current time.

Other annotation data that may be stored within the image table 308 isso-called “lens” data. A “lens” may be a real-time special effect andsound that may be added to an image or a video.

As mentioned above, the video table 310 stores video data which, in oneembodiment, is associated with messages for which records are maintainedwithin the message table 314. Similarly, the image table 308 storesimage data associated with messages for which message data is stored inthe message table 314. The entity table 302 may associate variousannotations from the annotation table 312 with various images and videosstored in the image table 308 and the video table 310.

A story table 306 stores data regarding collections of messages andassociated image, video, or audio data, which are compiled into acollection (e.g., a story or a gallery). The creation of a particularcollection may be initiated by a particular user (e.g., each user forwhom a record is maintained in the entity table 302). A user may createa “personal story” in the form of a collection of content that has beencreated and sent/broadcast by that user. To this end, the user interfaceof the messaging client application 104 may include an icon that isuser-selectable to enable a sending user to add specific content to hisor her personal story.

A collection may also constitute a “live story”, which is a collectionof content from multiple users that is created manually, automatically,or using a combination of manual and automatic techniques. For example,a “live story” may constitute a curated stream of user-submitted contentfrom various locations and events. Users whose client devices 102 havelocation services enabled and are at a common location or event at aparticular time may, for example, be presented with an option, via auser interface of the messaging client application 104, to contributecontent to a particular live story. The live story may be identified tothe user by the messaging client application 104, based on his or herlocation. The end result is a “live story” told from a communityperspective.

A further type of content collection is known as a “location story”,which enables a user whose client device 102 is located within aspecific geographic location (e.g., on a college or university campus)to contribute to a particular collection. In some embodiments, acontribution to a location story may require a second degree ofauthentication to verify that the end user belongs to a specificorganization or other entity (e.g., is a student on the universitycampus).

FIG. 4 is a schematic diagram illustrating a structure of a message 400,according to some embodiments, generated by a messaging clientapplication 104 for communication to a further messaging clientapplication 104 or the messaging server application 114. The content ofa particular message 400 is used to populate the message table 314stored within the database 120, accessible by the messaging serverapplication 114. Similarly, the content of a message 400 is stored inmemory as “in-transit” or “in-flight” data of the client device 102 orthe application server 112. The message 400 is shown to include thefollowing components:

-   -   A message identifier 402: a unique identifier that identifies        the message 400.    -   A message text payload 404: text, to be generated by a user via        a user interface of the client device 102, and that is included        in the message 400.    -   A message image payload 406: image data, captured by a camera        component of a client device 102 or retrieved from memory of a        client device 102, and that is included in the message 400.    -   A message video payload 408: video data, captured by a camera        component or retrieved from a memory component of the client        device 102, and that is included in the message 400.    -   A message audio payload 410: audio data, captured by a        microphone or retrieved from the memory component of the client        device 102, and that is included in the message 400.    -   Message annotations 412: annotation data (e.g., filters,        stickers, or other enhancements) that represents annotations to        be applied to the message image payload 406, message video        payload 408, or message audio payload 410 of the message 400.    -   A message duration parameter 414: a parameter value indicating,        in seconds, the amount of time for which content of the message        400 (e.g., the message image payload 406, message video payload        408, and message audio payload 410) is to be presented or made        accessible to a user via the messaging client application 104.    -   A message geolocation parameter 416: geolocation data (e.g.,        latitudinal and longitudinal coordinates) associated with the        content payload of the message 400. Multiple message geolocation        parameter 416 values may be included in the payload, each of        these parameter values being associated with respective content        items included in the content (e.g., a specific image in the        message image payload 406, or a specific video in the message        video payload 408).    -   A message story identifier 418: identifier values identifying        one or more content collections (e.g., “stories”) with which a        particular content item in the message image payload 406 of the        message 400 is associated. For example, multiple images within        the message image payload 406 may each be associated with        multiple content collections using identifier values.    -   A message tag 420: one or more tags, each of which is indicative        of the subject matter of content included in the message        payload. For example, where a particular image included in the        message image payload 406 depicts an animal (e.g., a lion), a        tag value may be included within the message tag 420 that is        indicative of the relevant animal. Tag values may be generated        manually, based on user input, or may be automatically generated        using, for example, image recognition.    -   A message sender identifier 422: an identifier (e.g., a        messaging system identifier, email address, or device        identifier) indicative of a user of the client device 102 on        which the message 400 was generated and from which the message        400 was sent.    -   A message receiver identifier 424: an identifier (e.g., a        messaging system identifier, email address, or device        identifier) indicative of a user of the client device 102 to        which the message 400 is addressed.

The contents (e.g., values) of the various components of the message 400may be pointers to locations in tables within which content data valuesare stored. For example, an image value in the message image payload 406may be a pointer to (or address of) a location within the image table308. Similarly, values within the message video payload 408 may point todata stored within the video table 310, values stored within the messageannotations 412 may point to data stored in the annotation table 312,values stored within the message story identifier 418 may point to datastored in the story table 306, and values stored within the messagesender identifier 422 and the message receiver identifier 424 may pointto user records stored within the entity table 302.

FIG. 5 is a schematic diagram illustrating an access-limiting process500, in terms of which access to content (e.g., an ephemeral message502, and associated multimedia payload of data) or a content collection(e.g., an ephemeral message story 504) may be time-limited (e.g., madeephemeral).

An ephemeral message 502 is shown to be associated with a messageduration parameter 506, the value of which determines an amount of timethat the ephemeral message 502 will be displayed to a receiving user ofthe ephemeral message 502 by the messaging client application 104. Inone embodiment, where the messaging client application 104 is anapplication client, an ephemeral message 502 is viewable by a receivinguser for up to a maximum of 10 seconds, depending on the amount of timethat the sending user specifies using the message duration parameter506.

The message duration parameter 506 and the message receiver identifier424 are shown to be inputs to a message timer 512, which is responsiblefor determining the amount of time that the ephemeral message 502 isshown to a particular receiving user identified by the message receiveridentifier 424. In particular, the ephemeral message 502 will only beshown to the relevant receiving user for a time period determined by thevalue of the message duration parameter 506. The message timer 512 isshown to provide output to a more generalized ephemeral timer system202, which is responsible for the overall timing of display of content(e.g., an ephemeral message 502) to a receiving user.

The ephemeral message 502 is shown in FIG. 5 to be included within anephemeral message story 504 (e.g., a personal story, or an event story).The ephemeral message story 504 has an associated story durationparameter 508, a value of which determines a time duration for which theephemeral message story 504 is presented and accessible to users of themessaging system 100. The story duration parameter 508, for example, maybe the duration of a music concert, where the ephemeral message story504 is a collection of content pertaining to that concert.Alternatively, a user (either the owning user or a curator user) mayspecify the value for the story duration parameter 508 when performingthe setup and creation of the ephemeral message story 504.

Additionally, each ephemeral message 502 within the ephemeral messagestory 504 has an associated story participation parameter 510, a valueof which determines the duration of time for which the ephemeral message502 will be accessible within the context of the ephemeral message story504. Accordingly, a particular ephemeral message 502 may “expire” andbecome inaccessible within the context of the ephemeral message story504, prior to the ephemeral message story 504 itself expiring in termsof the story duration parameter 508. The story duration parameter 508,story participation parameter 510, and message receiver identifier 424each provide input to a story timer 514, which operationally determineswhether a particular ephemeral message 502 of the ephemeral messagestory 504 will be displayed to a particular receiving user and, if so,for how long. Note that the ephemeral message story 504 is also aware ofthe identity of the particular receiving user as a result of the messagereceiver identifier 424.

Accordingly, the story timer 514 operationally controls the overalllifespan of an associated ephemeral message story 504, as well as anindividual ephemeral message 502 included in the ephemeral message story504. In one embodiment, each and every ephemeral message 502 within theephemeral message story 504 remains viewable and accessible for a timeperiod specified by the story duration parameter 508. In a furtherembodiment, a certain ephemeral message 502 may expire, within thecontext of the ephemeral message story 504, based on a storyparticipation parameter 510. Note that a message duration parameter 506may still determine the duration of time for which a particularephemeral message 502 is displayed to a receiving user, even within thecontext of the ephemeral message story 504. Accordingly, the messageduration parameter 506 determines the duration of time that a particularephemeral message 502 is displayed to a receiving user, regardless ofwhether the receiving user is viewing that ephemeral message 502 insideor outside the context of an ephemeral message story 504.

The ephemeral timer system 202 may furthermore operationally remove aparticular ephemeral message 502 from the ephemeral message story 504based on a determination that it has exceeded an associated storyparticipation parameter 510. For example, when a sending user hasestablished a story participation parameter 510 of 24 hours fromposting, the ephemeral timer system 202 will remove the relevantephemeral message 502 from the ephemeral message story 504 after thespecified 24 hours. The ephemeral timer system 202 also operates toremove an ephemeral message story 504 either when the storyparticipation parameter 510 for each and every ephemeral message 502within the ephemeral message story 504 has expired, or when theephemeral message story 504 itself has expired in terms of the storyduration parameter 508.

In certain use cases, a creator of a particular ephemeral message story504 may specify an indefinite story duration parameter 508. In thiscase, the expiration of the story participation parameter 510 for thelast remaining ephemeral message 502 within the ephemeral message story504 will determine when the ephemeral message story 504 itself expires.In this case, a new ephemeral message 502, added to the ephemeralmessage story 504, with a new story participation parameter 510,effectively extends the life of an ephemeral message story 504 to equalthe value of the story participation parameter 510.

In response to the ephemeral timer system 202 determining that anephemeral message story 504 has expired (e.g., is no longer accessible),the ephemeral timer system 202 communicates with the messaging system100 (e.g., specifically, the messaging client application 104) to causean indicium (e.g., an icon) associated with the relevant ephemeralmessage story 504 to no longer be displayed within a user interface ofthe messaging client application 104. Similarly, when the ephemeraltimer system 202 determines that the message duration parameter 506 fora particular ephemeral message 502 has expired, the ephemeral timersystem 202 causes the messaging client application 104 to no longerdisplay an indicium (e.g., an icon or textual identification) associatedwith the ephemeral message 502.

FIG. 6 illustrates example functional components of a pictorial symbolsystem 150, according to some example embodiments. As illustrated, thepictorial symbol system 150 comprises an interface engine 605, a tasksystem 610, a multimodal system 615, a content engine 620, and a displayengine 625. The interface engine 605 is configured to interface with theapplication 104. For example, the interface engine 605 can receivelocations of one or more images generated by the messaging clientapplication 104 using an image sensor of the client device 102. Further,according to some example embodiments, the interface engine 605 isconfigured to receive one or more text characters (e.g., a caption, asentence, one or more punctuation marks) for input into the pictorialsymbol system 150.

The task system 610 is a subsystem that is configured to generate one ormore pictorial symbol suggestions using a multi-task machine learningscheme, such as a multi-task neural network, as discussed in furtherdetail below with reference to FIGS. 7-10. The multimodal system 615 isa subsystem that is configured to generate one or more pictorial symbolsuggestions using a keyword speech recognition scheme, as discussed infurther detail below with reference to FIGS. 11-13. The content engine620 is configured to select one or more display elements that have beenpre-associated with categories or individual pictorial symbolpredictions generated by the task system 610 or the multimodal system615, as discussed in further detail below with reference to FIG. 14. Thedisplay engine 625 is configured to display the selected content on adisplay device of the client device 102. For example, the display engine625 may display the selected content in a suggestion window as a user isinputting text into the client device 102.

FIG. 7 illustrates example functional engines of the task system 610,according to some example embodiments. As illustrated, the task system610 comprises an interface engine 705, a training engine 710, a clusterengine 715, and a multi-task engine 720. The interface engine 705manages identifying text data for input into the multi-task engine 720and selecting content for display. The training engine 710 is configuredto train the machine learning schemes in the multi-task engine 720 ontraining data, such as network posts comprising one or more symbols(e.g., emojis) and text (e.g., captions). In some example embodiments,the training engine 710 is not integrated into a task system 610installed on the client device 102. Rather, in those exampleembodiments, the training engine 710 executes on the application server112 to generate machine learning models, such as neural network models,which can then be transmitted to client devices 102 for execution andprediction of symbols.

The cluster engine 715 is configured to generate clusters of pictorialsymbol categories. In some example embodiments, the cluster engine 715uses clustering schemes, such as K-means clustering, to semanticallygenerate classes of symbols that are semantically similar to oneanother.

The multi-task engine 720 is configured to implement a machine learningscheme that generates classifications or likelihoods that the subjectmatter of text input into the machine learning scheme corresponds topictorial symbols or categories of pictorial symbols (e.g., emojis andcategories of emojis).

FIG. 8A shows a flow diagram of a method 800 for pictorial symbolclassification and content selection, according to some exampleembodiments. At operation 805, the training engine 710 identifiestraining data. For example, at operation 805, the training engine 710may identify a collection of network posts (e.g., ephemeral message 502)that have a text caption and at least one pictorial symbol. In someexample embodiments, each of the network posts that have a caption andat least one pictorial symbol are used to generate skip-gram wordembeddings, which are used to train the multi-task neural network. Insome example embodiments, of the identified network posts, only poststhat contain the top N used emojis are stored as training data. Forexample, of the identified publicly available network posts, only poststhat include one of the top 20 user-selected emojis are identified andstored as training data at operation 805.

At operation 810, the training engine 710 trains the multi-task model ofa machine learning scheme (e.g., a neural network) for implementation inthe multi-task engine 720. In some example embodiments, the multi-taskmodel is trained end-to-end using neural network techniques (e.g., backpropagation, etc.) to maximize the likelihood of text input into themodel yielding a correct pictorial symbol output of a given text/symbolpair in the training data. Further details of training the multi-taskengine 720 are discussed in detail with reference to FIG. 10 below.

At operation 815, the multi-task engine 720 identifies text as inputdata. For example, the text can be characters input into the clientdevice 102 using a touch-screen keyboard. At operation 820, themulti-task engine 720 analyzes the text as it is being entered and usesthe texts to generate multi-task classifications using a machinelearning scheme trained on the model generated by the training engine710.

In some example embodiments, the multi-task classifications generated bythe multi-task engine 720 at operation 820 include a first likelihoodthat the subject matter of the text is correlated with individualpictorial symbols, a second likelihood that the subject matter of thetext is correlated with categories of pictorial symbols, and a thirdlikelihood that the subject matter of the text is correlated withsubcategories of pictorial symbols. Each of the classifications arenumerical likelihoods that a given pictorial symbol, category, orsubcategory is of a subject matter that is the same or similar to thesubject matter of the text being input by the user.

At operation 825, the interface engine 705 selects one or more displayelements or items of content based on the classifications generated atoperation 820. For example, if the multi-task engine 720 generates aclassification that the text is strongly correlated with an item offood, then at operation 825, the interface engine 705 selects a foodemoji as a suggestion for display.

In some example embodiments, the suggested emoji, emoji categories, andemoji sub-categories are used to identify non-emoji content that can besuggested to a user for inclusion in an ephemeral post. For instance,although the multi-task engine 720 outputs a heart emoji result, theheart emoji has been pre-associated with other love-related content(e.g., a cartoon cupid) that can be suggested for overlay in anephemeral message. Likewise, if a food emoji category is output of themulti-task engine 720, then a plurality of non-emoji food content items(e.g., cartoon taco, cartoon captions, video effects) can be suggestedto the user or otherwise displayed on the client device 102.

At operation 830, the display engine 625 displays the selected contenton a display device of the client device 102. The user may select thedisplayed content for inclusion in an ephemeral message, which is thenpublished to a network site for network access by other users throughthe application server 112.

FIG. 8B shows an example flow diagram of a method 835 for generatingclassifications, according to some example embodiments. Method 835 canbe configured as a subroutine of operation 825 in which one or moreclassifications are selected from the group of classifications generatedby the multi-task engine 720. At operation 845, the interface engine 705identifies the classification data output by the multi-task engine 720.The classification data can include single symbol classification data(denoted as “SS” in FIG. 8B), subcategory classification data (denotedas “SUB” in FIG. 8B), and category classification data (denoted as “CAT”in FIG. 8B). Each set of the classification data is compared against athreshold to determine whether the classification data is higher than aprespecified threshold. If a given set of classification data is higherthan its threshold, the classification is considered sufficientlyaccurate and the corresponding pictorial symbols can be included in thereturn set for suggestion to a user.

In particular, at operation 850, the interface engine 705 determineswhether the single symbol classification data satisfies a single symbolthreshold (“S-LVL”). If the single symbol classification data satisfiesthe threshold, content or pictorial symbols corresponding to the singlesymbol classification data are added to the return set at operation 865.

Likewise at operation 855, the interface engine 705 determines whetherthe subcategory classification data satisfies a subcategoryclassification data (“SUB-LVL”). If the subcategory classification datasatisfies the threshold, content or pictorial symbols corresponding tothe subcategory classification data are added to the return set atoperation 865.

Further, at operation 860, the interface engine 705 determines whetherthe category classification data satisfies a category classificationdata (“CAT-LVL”). If the category classification data satisfies thethreshold, content or pictorial symbols corresponding to the categoryclassification data are added to the return set at operation 865.

At operation 870, the interface engine 705 displays the content on adisplay device of the client device 102 as suggestions of content toinclude in an ephemeral message.

In some example embodiments, the different thresholds of the differentclassification sets are different values. For example, the confidence orthreshold for the single set may be higher than the subcategory categorythresholds to ensure that if a single symbol is suggested, it is morelikely to be an accurate related symbol.

In some example embodiments, not all classification types andcorresponding contents can be added to the return set. For example, thesingle pictorial symbol classification data can first be analyzed todetermine whether it satisfies the threshold. If the single pictorialsymbol classification data satisfies the threshold, then contentassociated with the single symbol is suggested to the user withoutanalyzing the subcategory and category level classifications. On theother hand, if the single pictorial symbol classification does notsatisfy its threshold, then as a fallback the subcategory classificationdata is analyzed to determine whether subcategory classification datameets the subcategory threshold. If the subcategory classification meetsits threshold, the content associated with the subcategory is displayedas suggestions to the user. On the other hand, if the subcategoryclassification does not meet the subcategory threshold, then the evenbroader category classification data is analyzed to determine whether itmeets the category level threshold. If the category classification datameets the category level threshold, the content associated with thecategory is displayed as suggestions to the user. In this way, if a morenarrow or specific item of content (e.g., an individual emoji) has ahigher score, competition or resources are conserved by displaying onlythe specific item of content without showing multiple items of contentof the broader subcategory and broadest category set.

FIG. 8C shows an example flow diagram of a method 875 for generatingclassifications, according to some example embodiments. Method 875 canbe configured as a subroutine of operation 825 in which one or moreclassifications are selected from the group of classifications generatedby the multi-task engine 720. As illustrated in the example of FIG. 8C,the subcategory and category level likelihoods and associated contentare replaced by a new category of pictorial symbols generated usingclustering techniques as discussed below, with reference to FIG. 9.

At operation 880, the interface engine 705 identifies the classificationdata output by the multi-task engine 720. The classification data caninclude single symbol classification data (denoted as “SS” in FIG. 8C)and cluster classification data (denoted as “CLUST.” in FIG. 8C). Eachset of the classification data is compared against a threshold todetermine whether the classification data is higher than a prespecifiedthreshold. If a given set of classification data is higher than itsthreshold, the classification is considered sufficiently accurate to theuser's caption and the corresponding pictorial symbols can be includedin the return set for suggestion to a user.

In particular, at operation 883, the interface engine 705 determineswhether the single symbol classification data satisfies a single symbolthreshold (“S-LVL”). If the single symbol classification data satisfiesthe threshold, content or pictorial symbols corresponding to the singlesymbol classification data are added to the return set at operation 890.

Likewise at operation 885, the interface engine 705 determines whetherthe cluster classification data satisfies a cluster classificationthreshold (“C-LVL”). If the cluster classification data satisfies thethreshold, content or pictorial symbols corresponding to the clusterclassification data are added to the return set at operation 890.

In some example embodiments, the different thresholds of the differentclassification sets are different values. For example, the confidence orthreshold for the single set may be higher than the cluster categorythreshold to ensure that if a single symbol is suggested, it is morelikely to be an accurate related symbol.

In some example embodiments, not all classification types andcorresponding contents are added to the return set. For example, thesingle pictorial symbol classification data can first be analyzed todetermine whether it satisfies the threshold. If the single pictorialsymbol classification data satisfies the threshold, then contentassociated with the single symbol is suggested to the user withoutanalyzing the cluster-based classifications. On the other hand, if thesingle pictorial symbol classification does not satisfy its threshold,then as a fallback the cluster classification data is analyzed todetermine whether cluster classification data meets the cluster categorythreshold. If the cluster category classification data meets itsthreshold, the content associated with the given cluster category isdisplayed as suggestions to the user. In this way, if a narrower or morespecific item of content (e.g., an individual emoji) has a higher score,competition or resources are conserved by displaying only the specificitem of content without showing multiple items of content of the broadercluster category.

FIG. 9 shows a flow diagram of a method 900 of semantically generatingpictorial symbols using a clustering scheme, according to some exampleembodiments. The cluster-based categories can be used as one type ofcategory for pictorial symbol suggestions, as discussed in FIG. 8Cabove. In some approaches, categories and sub-categories of emojis maypartition similar emojis into different categories even though they arerelated and commonly used together or as replacements in user posts. Forexample, referring to FIG. 17, emoji 1714 (“tired face”) and emoji 1740(“winking face with tongue”) are both in the Unicode sub-category ofneutral faces even though they clearly indicate different emotions.Another example: emoji 1704 (“smiling face with heart-eyes”) and emoji1706 (“red heart”) are semantically similar, but they appear indifferent Unicode categories. A more subtle example: emoji 1706 (“redheart”) and emoji 1718 (“blue heart”) are in the same Unicode categoryand in the same Unicode subcategory, but may be used by users indrastically different ways. For instance, a user that is sending a textmessage to an acquaintance may prefer the emoji 1718 (“blue heart”), andaccidently sending emoji 1706 (“red heart”) may result in amiscommunication.

To this end, a cluster engine 715 is configured to generate semanticallysimilar emojis into clusters, which can be treated as differentcategories (e.g., a semantically generated love category, a semanticallygenerated happy category, etc.) for training and symbol suggestions.With reference to FIG. 9, at operation 905, the cluster engine 715identifies training data. In some example embodiments, the training dataincludes publicly available posts (e.g., social media posts) thatcontain text and at least one pictorial symbol or emoji. In some exampleembodiments, of the available posts, only posts that contain the top Nused pictorial symbols are saved as training data. For example, whilethere may be hundreds of emojis in thousands of posts, only postscontaining the top 300 most frequently used emojis are stored astraining data. Generally, the semantically based categories enable alarger set of most frequently used emojis (e.g., 300) to be included inthe training set of data instead of smaller, more frequently used emojitraining sets (e.g., only including posts that contain the top 20 mostfrequently used emojis as training data).

At operation 910, the cluster engine 715 generates embeddings for eachof the pictorial symbols based on their accompanying text or caption. Insome example embodiments, the cluster engine 715 generates theembedding's using in a skip-gram based model. At operation 915, thecluster engine 715 clusters the generated embeddings using a clusteringscheme, such as K-means clustering. At operation 920, the cluster engine715 stores the semantically generated cluster categories for use in amulti-task neural network engine, such as multi-task engine 720.

FIG. 10 shows an example neural network architecture 1000 of themulti-task engine 720, according to some example embodiments. Asillustrated, input data 1002 (e.g., a caption, text sentence, one ormore characters of text) is used to generate character embedding data1006 from each character (e.g., a letter, a punctuation mark, a number)in the input data, and further used to generate word embedding data 1004from each word in the input data (e.g., each word in a caption orsentence accompanying an emoji). The character embedding data 1006 isinput into a character level long short-term memory (LSTM) 1008. In someexample embodiments, the LSTMs are bidirectional LSTMs, where aindividual bidirectional LSTM consists of a forward LSTM that processesinput data from left to right, and a backward LSTM that processes datafrom right to left. As a result, the output representation of abidirectional LSTM is an encoding in both time directions of a giveninput message.

The word embedding data 1004 and the output of the character LSTM 1008are fused together by the feature attention network 1010, according tosome example embodiments. The feature attention network 1010 isconfigured to linearly use multiple input signals instead of simplyconcatenating them. In some example embodiments, the feature attentionnetwork 1010 is trained using a unified word representation space, i.e.,feature attention network 1010 generates a single vector representationwith aggregated knowledge among the multiple input word representationsbased on their weighted importance.

Sentence embeddings are computed using a stack of bidirectional LSTMsand a word attention layer. The word attention layer is configured toplace emphasis or weight on words that are more important in a givensentence or caption. With reference to FIG. 10, the output of featureattention network 1010 is an input into a first word LSTM network 1012,which outputs into a second word LSTM network 1014. In some exampleembodiments, the architecture 1000 omits the feature attention network1010 and the character embedding data 1006, and the output of thecharacter LSTM 1008 is input directly into the first word LSTM network1012. Further, the output of the first word LSTM network 1012 and thesecond word LSTM network 1014 is input into a word attention network1016.

The output of the word attention network 1016 is input into classifierpairs, such as a fully connected network that outputs into a SoftMaxlayer. In particular, as illustrated in FIG. 10, word attention network1016 outputs representation data into a plurality of fully connectednetworks, such as fully connected network 1018, fully connected network1022, and fully connected network 1026. Each of the fully connectednetworks inputs into a respective SoftMax layer: fully connected network1018 inputs into SoftMax layer 1020, fully connected network 1022 inputsinto SoftMax layer 1024, and fully connected network 1026 inputs intoSoftMax layer 1028.

Each classifier pair is trained for a different task. For example, thefully connected network 1018 and the SoftMax layer 1020 may be trainedfor a first task, such as single symbol prediction. Further, the fullyconnected network 1022 and its corresponding SoftMax layer 1024 may betrained for a second task, such as subcategory level symbol prediction.Further, the fully connected network 1026 and its corresponding SoftMaxlayer 1028 may be trained for a third task, such as category levelsymbol prediction. In some example embodiments, the quantity of taskssets how many fully connected layers and classification layer pairs areincluded in architecture 1000. For example, to implement method 875(FIG. 8C) in which only two tasks are used (e.g., a first single symbolprediction task, and a second semantically clustered category predictiontask), only two sets of fully connected layers and classification layersare used (i.e., fully connected network 1026 and SoftMax layer 1028would be removed from architecture 1000).

In some example embodiments, parameters are shared across the entirenetwork architecture 1000, and specialization for different tasks onlyoccurs at the final stage (e.g., in which the fully connected neuralnetworks output to respective SoftMax layers) to predict specific labelsfor each different task. In some example embodiments the specializationis implemented using linear transformations and cross entropy lossfunction for all classification tasks. In some embodiments, the finalloss of architecture 1000 is the sum of each single loss for each of thetasks.

FIG. 11 shows example internal functional engines of a multimodal system615, according to some example embodiments. As illustrated, themultimodal system 615 comprises a text filter engine 1100, a musicfilter engine 1105, a language filter engine 1110, a keyword engine1115, and a classification engine 1120. To avoid obscuring the inventivesubject matter with unnecessary detail, various functional components(e.g., modules and engines) that are not germane to conveying anunderstanding of the inventive subject matter have been omitted fromFIG. 11. However, a skilled artisan will readily recognize that variousadditional functional components may be supported by the multimodalsystem 615 to facilitate additional functionality that is notspecifically described herein.

As is understood by skilled artisans in the relevant computer arts, eachfunctional component (e.g., module) illustrated in FIG. 11 may beimplemented using hardware (e.g., a processor of a machine) or acombination of logic (e.g., executable software instructions) andhardware (e.g., memory and processor of a machine) for executing thelogic. Furthermore, the various functional components depicted in FIG.11 may reside on a single computer (e.g., a laptop), or may bedistributed across several computers in various arrangements such ascloud-based architectures. Moreover, any two or more modules of themultimodal system 615 may be combined into a single module, orsubdivided among multiple modules. It shall be appreciated that whilethe functional components (e.g., modules) of FIG. 11 are discussed inthe singular sense, in other embodiments, multiple instances of one ormore of the modules may be employed.

The text filter engine 1100 is configured to retrieve training data(e.g., publicly available social network posts) and filter out one ormore of the retrieved items based on whether they have at least aprespecified quantity of pictorial symbols. The music filter engine 1105is configured to detect items in the training data that have audio datafeaturing music (e.g., a video clip of a user with music playing in thebackground). The music filter engine 1105 is configured to remove orotherwise filter the detected items from the training data. The languagefilter engine 1110 is configured to remove or otherwise filter items inthe training data that comprise words spoken in a language other thanthe primary language. In some example embodiments, the language filterengine 1110 identifies the primary language using the keyboard settingof the client device 102 of the user, or by implementing the machinelearning scheme. The keyword engine 1115 is configured to implement akeyword recognition scheme to identify keywords in audio data in thereceived input data (e.g., an ephemeral message to be published). Theclassification engine 1120 is configured to receive input data and use atrained machine learning scheme (e.g., random forest) to generate apictorial symbol suggestion.

FIG. 12 shows a flow diagram of a method 1200 for implementing emojisuggestions from multimodal messages, according to some exampleembodiments. At operation 1205, the text filter engine 1100 retrievespublicly available messages or posts (e.g., posts available over network106 to the general public). The messages can include recently publishedmultimodal messages having image or video, audio data, overlaidcaptions, emojis, and/or other visual overlaid elements.

At operation 1210, the text filter engine 1100 removes or otherwisefilters messages that do not have at least one pictorial symbol (e.g.,at least one emoji). At operation 1215, the pictorial symbol system 150removes or otherwise filters messages that do not have voice audio data(e.g., images, video clips with no audio data). At operation 1220, themusic filter engine 1105 removes messages that have music audio data.For example, some of the messages may be multimodal messages includingvideo data and audio data with music playing in the background. Themusic data can degrade the keyword recognition process. The music filterengine 1105 is trained to detect music in snaps and then remove orotherwise filter out the messages that include music. The music filterengine 1105 is trained with SoundNet features and a Deep Neural Network(DNN) is implemented to model SoundNet features for a binary (e.g., 0/1)classification. In some embodiments, a dataset that contains manuallylabeled music/non-music information is used to train the DNN model. Insome embodiments, the DNN model has four fully connected layers, witheach layer comprising 512 hidden neurons. The music filter engine 1105implements majority voting to consolidate the frame level results intoutterance level. The music filter engine 1105 then picks layer of conv4from SoundNet as the features to use.

At operation 1225, the language filter engine 1110 removes messages thathave audio voice data in a non-primary language. The machine learningsystems are typically configured or trained for a certain language for agiven geographical area. In some example embodiments, a language that ismost predominant in the geographic area is set as the primary language,and the language filter engine 1110 removes messages that have voicedata in a non-primary language for that region or demographic to improvesuggestions for users in that region or demographic.

In some embodiments, the primary language is identified by determiningwhat the keyboard language setting is of the user's client device (e.g.,of a client device executing the multimodal system 615). In some exampleembodiments, the language filter engine 1110 implements a languagemachine learning scheme (e.g., langId.py) that learns large numbers ofoverlapping words and character-based n-gram features.

FIG. 13 shows a flow diagram of a method 1300 for multimodalsuggestions, according to some example embodiments. The method 1300 maybe implemented by the multimodal system 615 after the method 1200. Atoperation 1305, the keyword engine 1115 implements a speech recognizerto identify keywords in the retrieved messages. For example, atoperation 1305, the keyword engine 1115 uses an Automatic SpeechRecognition system using a modified Kaldi system, with forty-dimensionMel filter bank features, and a chain model for acoustic modeling. Thetraining acoustic data includes publicly available datasets and someproprietary datasets. The proprietary datasets consist of short audiorecordings with various types of background noise. The speech in thedataset is mainly spontaneous speech in casual talking style. Thelanguage model is interpolated from a general language model and alanguage model trained on the proprietary dataset.

At operation 1310, the keyword engine 1115 implements a keyword spotterto identify additional keywords in the messages. For example, atoperation 1310, the keyword engine 1115 implements a neural networkconfigured to recognize specific keywords. In some embodiments, thekeywords used to train the neural network are spoken words in thepublicly available multimodal messages. In some embodiments, thekeywords used to train the neural network are from a pre-set list. Anexample pre-set list of keywords includes: America, american,announcement, ball, black, breakfast, celebration, climax, cold,congratulations, cool, crazy, cry, crying, death, delicious, desire,dessert, donut, doughnut, eat, emotion, evil, exam, exhausted, finals,fire, flame, flirt, food, football, freezing, fried, frustrated,glasses, goofy, gridiron, grin, grinning, happy, heart, hearts, homer,hot, hundred, hungry, hurt, intense, joint, joy, kidding, kiss, kissing,laughing, lips, love, loyalty, marijuana, melancholy, morn, ok, okay,party, passion, pastry, percent, perfect, pink, playful, plus, police,popper, pot, quiz, romance, sad, savoring, savory, school, score, silly,sleepy, smile, smoke, smooth, sob, sobbing, somber, sport, stability,study, sun, sunglasses, sunny, superbowl, sweet, tada, tasty, tears,test, tired, tiredness, tongue, trust, truth, victory, weary, wink,winking, and yummy.

At operation 1315, the classification engine 1120 classifies thekeywords into categories. In some example embodiments, theclassification engine 1120 implements a bag-of-words (BOW) classifier toclassify keywords into categories at operation 1315. The bag-of-wordsclassifier represents text as a vector that includes such features asterm frequency inverse document frequency (TF-IDF). The BOW modelfeature set can be expanded with bigrams and trigrams.

At operation 1320, the classification engine 1120 trains a machinelearning model on the keywords that have been classified intocategories. In some example embodiments, the classification engine 1120trains a random forest model with 100 trees (sklearn implementation) onthe keyword categories.

At operation 1325, the classification engine 1120 receives input data,such as a multimodal message (e.g., an ephemeral message of a userexclaiming into the camera “wow!” or “mind blown!”). In response toreceiving the message, the classification engine 1120 may automaticallyimplement the trained machine learning scheme, such as random forest, toanalyze the voice audio data (e.g., “wow”) to generate an emojisuggestion (e.g., emoji 1736 “mushroom cloud”) at operation 1330.

FIG. 14 shows an example flow diagram of a method 1400 for displayingsymbol results, according to some example embodiments. At operation1405, the content engine 620 identifies input data. For example, atoperation 1405, the content engine 620 identifies a message to bepublished to a network site. At operation 1410, the content engine 620determines whether the input data contains audio data (e.g., a videoclip with audio data). If the input data does not contain audio data,then the content engine 620 initiates the task system 610 to generatemulti-task based symbol suggestions at operation 1415. The resultsgenerated by the task system 610 are then displayed by the contentengine 620 as suggestions to the user at operation 1425. On the otherhand, if the input data does contain audio data, then the content engine620 initiates the task system 610 and the multimodal system 615 atoperation 1420 to generate symbol suggestions. At operation 1425 thesymbol or content suggestions generated by the different systems aredisplayed to the user as suggestions in different areas of a userinterface, as discussed below with reference to FIG. 16.

FIG. 15 shows an example user interface 1500 for pictorial symbolprediction, according to some example embodiments. In the example ofFIG. 15, the user interface 1500 displays a client-device-captured imagedepicting a singer 1505 on a stage that is singing to an audience 1510.The user of the client device has further input a caption 1515 using anon-screen keyboard (not depicted). In some example embodiments thecaption 1515 is input into the pictorial symbol system 150, whichoutputs suggested content, such as a heart sticker image 1525, in akeyboard area 1520 of the user interface 1500. The user then may selectthe heart sticker image 1525 to be included in the caption 1515 or asoverlay content anywhere in the image.

FIG. 16 shows an example user interface 1600 for pictorial symbolprediction, according to some example embodiments. In the example ofFIG. 16, the user interface 1600 displays a client-device-captured imagedepicting a taco 1605. A user of the client device has further input acaption 1610 using an on-screen keyboard (not depicted). The caption1610 is input into the pictorial symbol system 150, which outputssuggested content in a keyboard area 1615. For example, a first window1620 can display an individual symbol generated by the task system 610,a second window 1625 can display subcategories or categories of symbolsthat are generated by the task system 610, and further a third window1630 can display symbols from the multimodal system 615.

FIG. 17 shows an example set of pictorial symbols 1700, according tosome example embodiments. The pictorial symbols 1700 included are emojis1702-1740, each of which belongs to a Unicode category and subcategory.

FIG. 18 is a block diagram illustrating an example software architecture1806, which may be used in conjunction with various hardwarearchitectures herein described. FIG. 18 is a non-limiting example of asoftware architecture, and it will be appreciated that many otherarchitectures may be implemented to facilitate the functionalitydescribed herein. The software architecture 1806 may execute on hardwaresuch as a machine 1900 of FIG. 19 that includes, among other things,processors 1910, memory 1932, and I/O components 1950. A representativehardware layer 1852 is illustrated and can represent, for example, themachine 1900 of FIG. 19. The representative hardware layer 1852 includesa processing unit 1854 having associated executable instructions 1804.The executable instructions 1804 represent the executable instructionsof the software architecture 1806, including implementation of themethods, components, and so forth described herein. The hardware layer1852 also includes memory and/or storage modules 1856, which also havethe executable instructions 1804. The hardware layer 1852 may alsocomprise other hardware 1858.

In the example architecture of FIG. 18, the software architecture 1806may be conceptualized as a stack of layers where each layer providesparticular functionality. For example, the software architecture 1806may include layers such as an operating system 1802, libraries 1820,frameworks/middleware 1818, applications 1816, and a presentation layer1814. Operationally, the applications 1816 and/or other componentswithin the layers may invoke application programming interface (API)calls 1808 through the software stack and receive a response in the formof messages 1812. The layers illustrated are representative in natureand not all software architectures have all layers. For example, somemobile or special-purpose operating systems may not provide aframeworks/middleware 1818, while others may provide such a layer. Othersoftware architectures may include additional or different layers.

The operating system 1802 may manage hardware resources and providecommon services. The operating system 1802 may include, for example, akernel 1822, services 1824, and drivers 1826. The kernel 1822 may act asan abstraction layer between the hardware and the other software layers.For example, the kernel 1822 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 1824 may provideother common services for the other software layers. The drivers 1826are responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 1826 include display drivers, cameradrivers, Bluetooth® drivers, flash memory drivers, serial communicationdrivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers,audio drivers, power management drivers, and so forth depending on thehardware configuration.

The libraries 1820 provide a common infrastructure that is used by theapplications 1816 and/or other components and/or layers. The libraries1820 provide functionality that allows other software components toperform tasks in an easier fashion than by interfacing directly with theunderlying operating system 1802 functionality (e.g., kernel 1822,services 1824, and/or drivers 1826). The libraries 1820 may includesystem libraries 1844 (e.g., C standard library) that may providefunctions such as memory allocation functions, string manipulationfunctions, mathematical functions, and the like. In addition, thelibraries 1820 may include API libraries 1846 such as media libraries(e.g., libraries to support presentation and manipulation of variousmedia formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, or PNG),graphics libraries (e.g., an OpenGL framework that may be used to render2D and 3D graphic content on a display), database libraries (e.g.,SQLite that may provide various relational database functions), weblibraries (e.g., WebKit that may provide web browsing functionality),and the like. The libraries 1820 may also include a wide variety ofother libraries 1848 to provide many other APIs to the applications 1816and other software components/modules.

The frameworks/middleware 1818 provide a higher-level commoninfrastructure that may be used by the applications 1816 and/or othersoftware components/modules. For example, the frameworks/middleware 1818may provide various graphic user interface (GUI) functions, high-levelresource management, high-level location services, and so forth. Theframeworks/middleware 1818 may provide a broad spectrum of other APIsthat may be utilized by the applications 1816 and/or other softwarecomponents/modules, some of which may be specific to a particularoperating system 1802 or platform.

The applications 1816 include built-in applications 1838 and/orthird-party applications 1840. Examples of representative built-inapplications 1838 may include, but are not limited to, a contactsapplication, a browser application, a book reader application, alocation application, a media application, a messaging application,and/or a game application. The third-party applications 1840 may includean application developed using the ANDROID™ or IOS™ software developmentkit (SDK) by an entity other than the vendor of the particular platform,and may be mobile software running on a mobile operating system such asIOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. Thethird-party applications 1840 may invoke the API calls 1808 provided bythe mobile operating system (such as the operating system 1802) tofacilitate functionality described herein.

The applications 1816 may use built-in operating system functions (e.g.,kernel 1822, services 1824, and/or drivers 1826), libraries 1820, andframeworks/middleware 1818 to create user interfaces to interact withusers of the system. Alternatively, or additionally, in some systemsinteractions with a user may occur through a presentation layer, such asthe presentation layer 1814. In these systems, the application/component“logic” can be separated from the aspects of the application/componentthat interact with a user.

FIG. 19 is a block diagram illustrating components of a machine 1900,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.Specifically, FIG. 19 shows a diagrammatic representation of the machine1900 in the example form of a computer system, within which instructions1916 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 1900 to perform any oneor more of the methodologies discussed herein may be executed. As such,the instructions 1916 may be used to implement modules or componentsdescribed herein. The instructions 1916 transform the general,non-programmed machine 1900 into a particular machine 1900 programmed tocarry out the described and illustrated functions in the mannerdescribed. In alternative embodiments, the machine 1900 operates as astandalone device or may be coupled (e.g., networked) to other machines.In a networked deployment, the machine 1900 may operate in the capacityof a server machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1900 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a set-top box (STB), apersonal digital assistant (PDA), an entertainment media system, acellular telephone, a smartphone, a mobile device, a wearable device(e.g., a smart watch), a smart home device (e.g., a smart appliance),other smart devices, a web appliance, a network router, a networkswitch, a network bridge, or any machine capable of executing theinstructions 1916, sequentially or otherwise, that specify actions to betaken by the machine 1900. Further, while only a single machine 1900 isillustrated, the term “machine” shall also be taken to include acollection of machines that individually or jointly execute theinstructions 1916 to perform any one or more of the methodologiesdiscussed herein.

The machine 1900 may include processors 1910 having individualprocessors 1912 and 1914 (e.g., cores), memory/storage 1930, and I/Ocomponents 1950, which may be configured to communicate with each othersuch as via a bus 1902. The memory/storage 1930 may include a memory1932, such as a main memory, or other memory storage, and a storage unit1936, both accessible to the processors 1910 such as via the bus 1902.The storage unit 1936 and memory 1932 store the instructions 1916embodying any one or more of the methodologies or functions describedherein. The instructions 1916 may also reside, completely or partially,within the memory 1932, within the storage unit 1936, within at leastone of the processors 1910 (e.g., within the processor's cache memory),or any suitable combination thereof, during execution thereof by themachine 1900. Accordingly, the memory 1932, the storage unit 1936, andthe memory of the processors 1910 are examples of machine-readablemedia.

The I/O components 1950 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 1950 that are included in a particular machine 1900 willdepend on the type of machine. For example, portable machines such asmobile phones will likely include a touch input device or other suchinput mechanisms, while a headless server machine will likely notinclude such a touch input device. It will be appreciated that the I/Ocomponents 1950 may include many other components that are not shown inFIG. 19. The I/O components 1950 are grouped according to functionalitymerely for simplifying the following discussion and the grouping is inno way limiting. In various example embodiments, the I/O components 1950may include output components 1952 and input components 1954. The outputcomponents 1952 may include visual components (e.g., a display such as aplasma display panel (PDP), a light-emitting diode (LED) display, aliquid-crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), haptic components (e.g., avibratory motor, resistance mechanisms), other signal generators, and soforth. The input components 1954 may include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point-based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or other pointinginstruments), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

In further example embodiments, the I/O components 1950 may includebiometric components 1956, motion components 1958, environmentcomponents 1960, or position components 1962 among a wide array of othercomponents. For example, the biometric components 1956 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram-basedidentification), and the like. The motion components 1958 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environment components 1960 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gassensors to detect concentrations of hazardous gases for safety or tomeasure pollutants in the atmosphere), or other components that mayprovide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 1962 mayinclude location sensor components (e.g., a GPS receiver component),altitude sensor components (e.g., altimeters or barometers that detectair pressure from which altitude may be derived), orientation sensorcomponents (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1950 may include communication components 1964operable to couple the machine 1900 to a network 1980 or devices 1970via a coupling 1982 and a coupling 1972, respectively. For example, thecommunication components 1964 may include a network interface componentor other suitable device to interface with the network 1980. In furtherexamples, the communication components 1964 may include wiredcommunication components, wireless communication components, cellularcommunication components, near field communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 1970 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUniversal Serial Bus (USB)).

Moreover, the communication components 1964 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 1964 may include radio frequency identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional barcodes such as Universal Product Code (UPC) barcode,multi-dimensional barcodes such as Quick Response (QR) code, Aztec code,Data Matrix, Dataglyph, MaxiCode, PDF419, Ultra Code, UCC RSS-2Dbarcode, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components1964, such as location via Internet Protocol (IP) geolocation, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

Glossary

“CARRIER SIGNAL” in this context refers to any intangible medium that iscapable of storing, encoding, or carrying instructions for execution bythe machine, and includes digital or analog communications signals orother intangible media to facilitate communication of such instructions.Instructions may be transmitted or received over the network using atransmission medium via a network interface device and using any one ofa number of well-known transfer protocols.

“CLIENT DEVICE” in this context refers to any machine that interfaces toa communications network to obtain resources from one or more serversystems or other client devices. A client device may be, but is notlimited to, a mobile phone, desktop computer, laptop, PDA, smartphone,tablet, ultrabook, netbook, multi-processor system, microprocessor-basedor programmable consumer electronics system, game console, set-top box,or any other communication device that a user may use to access anetwork.

“COMMIUNICATIONS NETWORK” in this context refers to one or more portionsof a network that may be an ad hoc network, an intranet, an extranet, avirtual private network (VPN), a local area network (LAN), a wirelessLAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), ametropolitan area network (MAN), the Internet, a portion of theInternet, a portion of the Public Switched Telephone Network (PSTN), aplain old telephone service (POTS) network, a cellular telephonenetwork, a wireless network, a Wi-Fi® network, another type of network,or a combination of two or more such networks. For example, a network ora portion of a network may include a wireless or cellular network andthe coupling may be a Code Division Multiple Access (CDMA) connection, aGlobal System for Mobile communications (GSM) connection, or anothertype of cellular or wireless coupling. In this example, the coupling mayimplement any of a variety of types of data transfer technology, such asSingle Carrier Radio Transmission Technology (1×RTT), Evolution-DataOptimized (EVDO) technology, General Packet Radio Service (GPRS)technology, Enhanced Data rates for GSM Evolution (EDGE) technology,third Generation Partnership Project (3GPP) including 3G, fourthgeneration wireless (4G) networks, Universal Mobile TelecommunicationsSystem (UMTS), High-Speed Packet Access (HSPA), WorldwideInteroperability for Microwave Access (WiMAX), Long-Term Evolution (LTE)standard, others defined by various standard-setting organizations,other long-range protocols, or other data transfer technology.

“EPHEMERAL MESSAGE” in this context refers to a message that isaccessible for a time-limited duration. An ephemeral message 502 may bea text, an image, a video, and the like. The access time for theephemeral message 502 may be set by the message sender. Alternatively,the access time may be a default setting or a setting specified by therecipient. Regardless of the setting technique, the message istransitory.

“MACHINE-READABLE MEDIUM” in this context refers to a component, adevice, or other tangible media able to store instructions and datatemporarily or permanently and may include, but is not limited to,random-access memory (RAM), read-only memory (ROM), buffer memory, flashmemory, optical media, magnetic media, cache memory, other types ofstorage (e.g., erasable programmable read-only memory (EPROM)), and/orany suitable combination thereof. The term “machine-readable medium”should be taken to include a single medium or multiple media (e.g., acentralized or distributed database, or associated caches and servers)able to store instructions. The term “machine-readable medium” shallalso be taken to include any medium, or combination of multiple media,that is capable of storing instructions (e.g., code) for execution by amachine, such that the instructions, when executed by one or moreprocessors of the machine, cause the machine to perform any one or moreof the methodologies described herein. Accordingly, a “machine-readablemedium” refers to a single storage apparatus or device, as well as“cloud-based” storage systems or storage networks that include multiplestorage apparatus or devices. The term “machine-readable medium”excludes signals per se.

“COMPONENT” in this context refers to a device, a physical entity, orlogic having boundaries defined by function or subroutine calls, branchpoints, APIs, or other technologies that provide for the partitioning ormodularization of particular processing or control functions. Componentsmay be combined via their interfaces with other components to carry outa machine process. A component may be a packaged functional hardwareunit designed for use with other components and a part of a program thatusually performs a particular function of related functions. Componentsmay constitute either software components (e.g., code embodied on amachine-readable medium) or hardware components. A “hardware component”is a tangible unit capable of performing certain operations and may beconfigured or arranged in a certain physical manner.

In various example embodiments, one or more computer systems (e.g., astandalone computer system, a client computer system, or a servercomputer system) or one or more hardware components of a computer system(e.g., a processor or a group of processors) may be configured bysoftware (e.g., an application or application portion) as a hardwarecomponent that operates to perform certain operations as describedherein. A hardware component may also be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware component may include dedicated circuitry or logic that ispermanently configured to perform certain operations. A hardwarecomponent may be a special-purpose processor, such as afield-programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC). A hardware component may also includeprogrammable logic or circuitry that is temporarily configured bysoftware to perform certain operations. For example, a hardwarecomponent may include software executed by a general-purpose processoror other programmable processor. Once configured by such software,hardware components become specific machines (or specific components ofa machine) uniquely tailored to perform the configured functions and areno longer general-purpose processors.

It will be appreciated that the decision to implement a hardwarecomponent mechanically, in dedicated and permanently configuredcircuitry, or in temporarily configured circuitry (e.g., configured bysoftware) may be driven by cost and time considerations. Accordingly,the phrase “hardware component” (or “hardware-implemented component”)should be understood to encompass a tangible entity, be that an entitythat is physically constructed, permanently configured (e.g.,hardwired), or temporarily configured (e.g., programmed) to operate in acertain manner or to perform certain operations described herein.Considering embodiments in which hardware components are temporarilyconfigured (e.g., programmed), each of the hardware components need notbe configured or instantiated at any one instance in time. For example,where a hardware component comprises a general-purpose processorconfigured by software to become a special-purpose processor, thegeneral-purpose processor may be configured as respectively differentspecial-purpose processors (e.g., comprising different hardwarecomponents) at different times. Software accordingly configures aparticular processor or processors, for example, to constitute aparticular hardware component at one instance of time and to constitutea different hardware component at a different instance of time.

Hardware components can provide information to, and receive informationfrom, other hardware components. Accordingly, the described hardwarecomponents may be regarded as being communicatively coupled. Wheremultiple hardware components exist contemporaneously, communications maybe achieved through signal transmission (e.g., over appropriate circuitsand buses) between or among two or more of the hardware components. Inembodiments in which multiple hardware components are configured orinstantiated at different times, communications between or among suchhardware components may be achieved, for example, through the storageand retrieval of information in memory structures to which the multiplehardware components have access. For example, one hardware component mayperform an operation and store the output of that operation in a memorydevice to which it is communicatively coupled. A further hardwarecomponent may then, at a later time, access the memory device toretrieve and process the stored output.

Hardware components may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation). The various operations of example methods described hereinmay be performed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implementedcomponents that operate to perform one or more operations or functionsdescribed herein. As used herein, “processor-implemented component”refers to a hardware component implemented using one or more processors.Similarly, the methods described herein may be at least partiallyprocessor-implemented, with a particular processor or processors beingan example of hardware. For example, at least some of the operations ofa method may be performed by one or more processors orprocessor-implemented components.

Moreover, the one or more processors may also operate to supportperformance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines including processors), with these operationsbeing accessible via a network (e.g., the Internet) and via one or moreappropriate interfaces (e.g., an API). The performance of certain of theoperations may be distributed among the processors, not only residingwithin a single machine, but deployed across a number of machines. Insome example embodiments, the processors or processor-implementedcomponents may be located in a single geographic location (e.g., withina home environment, an office environment, or a server farm). In otherexample embodiments, the processors or processor-implemented componentsmay be distributed across a number of geographic locations.

“PROCESSOR” in this context refers to any circuit or virtual circuit (aphysical circuit emulated by logic executing on an actual processor)that manipulates data values according to control signals (e.g.,“commands”, “op codes”, “machine code”, etc.) and which producescorresponding output signals that are applied to operate a machine. Aprocessor may, for example, be a Central Processing Unit (CPU), aReduced Instruction Set Computing (RISC) processor, a ComplexInstruction Set Computing (CISC) processor, a Graphics Processing Unit(GPU), a Digital Signal Processor (DSP), an ASIC, a Radio-FrequencyIntegrated Circuit (RFIC), or any combination thereof. A processor mayfurther be a multi-core processor having two or more independentprocessors (sometimes referred to as “cores”) that may executeinstructions contemporaneously.

“TIMESTAMP” in this context refers to a sequence of characters orencoded information identifying when a certain event occurred, forexample giving date and time of day, sometimes accurate to a smallfraction of a second.

What is claimed is:
 1. A method comprising: identifying one or more text characters input into a client device; generating a plurality of pictorial symbol classifications from the one or more text characters using a machine learning scheme, the plurality of pictorial symbol classifications including at least one pictorial symbol category classification for a group of pictorial symbols of a similar type; determining that one of the plurality of pictorial symbol classifications satisfies a specified threshold; and displaying, on a display device of the client device, a presentation of one or more display elements that are pre-associated with the one of the plurality of pictorial symbol classifications that satisfies the specified threshold, wherein the machine learning scheme is a multitask neural network having a first network portion configured to generate individual pictorial symbol classifications and a second network portion configured to generate pictorial symbol category classifications.
 2. The method of claim 1, further comprising: determining that an individual pictorial symbol classification generated by the first network portion does not satisfy the specified threshold; and wherein the at least one pictorial symbol category classification is the one of the plurality of pictorial symbol classifications that satisfies the specified threshold.
 3. The method of claim 1, wherein the multitask neural network further comprises a third network portion configured to generate pictorial symbol subcategory classifications.
 4. The method of claim 3, wherein each pictorial symbol category classification corresponds to a category of pictorial symbols of a similar type, and wherein each pictorial symbol subcategory classification corresponds to a subcategory of the category of pictorial symbols of a more similar type.
 5. The method of claim 4, wherein categories and subcategories of the pictorial symbol classifications have been manually arranged by one or more users.
 6. The method of claim 1, wherein the machine learning scheme generates the plurality of pictorial symbol classifications in parallel.
 7. The method of claim 1, further comprising: generating categories of the pictorial symbol classifications using an additional machine learning scheme configured to cluster similar types of pictorial symbols.
 8. The method of claim 7, wherein the additional machine learning scheme implements k-means clustering to group similar types of pictorial symbols.
 9. The method of claim 8, wherein the additional machine learning scheme is trained using a plurality of network published posts, each of the plurality of network published posts comprising one or more text characters and a pictorial symbol.
 10. The method of claim 9, wherein each of the plurality of network published posts comprises at most one pictorial symbol.
 11. The method of claim 1, wherein the one or more display elements include one or more of: an emoji, an emoticon, an image, a sentence.
 12. The method of claim 1, wherein pictorial symbols are one or more of: an emoji or an image.
 13. A method comprising: identifying one or more text characters input into a client device; generating a plurality of pictorial symbol classifications from the one or more text characters using a machine learning scheme, the plurality of pictorial symbol classifications including at least one pictorial symbol category classification for a group of pictorial symbols of a similar type; determining that one of the plurality of pictorial symbol classifications satisfies a specified threshold; and displaying, on a display device of the client device, a presentation of one or more display elements that are pre-associated with the one of the plurality of pictorial symbol classifications that satisfies the specified threshold; generating one or more images on the client device; receiving the one or more text characters as a caption for overlay on the one or more images; and generating a symbol prediction using a random forest learning scheme, the random forest learning scheme trained on multimodal messages comprising voice data.
 14. A system comprising: one or more processors of a machine; and a memory storing instructions that, when executed by at least one processor among the one or more processors, causes the machine to perform operations comprising: identifying one or more text characters input into a client device; generating a plurality of pictorial symbol classifications from the one or more text characters using a machine learning scheme, the plurality of pictorial symbol classifications including at least one pictorial symbol category classification for a group of pictorial symbols of a similar type; determining that one of the plurality of pictorial symbol classifications satisfies a specified threshold; and displaying, on a display device of the client device, a presentation of one or more display elements that are pre-associated with the one of the plurality of pictorial symbol classifications that satisfies the specified threshold, wherein the machine learning scheme is a multitask neural network having a first network portion configured to generate individual pictorial symbol classifications and a second network portion configured to generate pictorial symbol category classifications.
 15. The system of claim 14, the operations further comprising: determining that an individual pictorial symbol classification generated by the first network portion does not satisfy the specified threshold; and wherein the at least one pictorial symbol category classification is the one of the plurality of pictorial symbol classifications that satisfies the specified threshold.
 16. A machine-readable storage device embodying instructions that, when executed by a device, cause the device to perform operations comprising: identifying one or more text characters input into a client device; generating a plurality of pictorial symbol classifications from the one or more text characters using a machine learning scheme, the plurality of pictorial symbol classifications including at least one pictorial symbol category classification for a group of pictorial symbols of a similar type, determining that one of the plurality of pictorial symbol classifications satisfies a specified threshold; and displaying, on a display device of the client device, a presentation of one or more display elements that are pre-associated with the one of the plurality of pictorial symbol classifications that satisfies the specified threshold, wherein the machine learning scheme is a multitask neural network having a first network portion configured to generate individual pictorial symbol classifications and a second network portion configured to generate pictorial symbol category classifications. 